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This report explores the scope arid limitations of the use of citations of 
published works as measures of the influential productivity of scientists, and 
suggests that publication citations be utilized for educational research. The Science 
Citation Index is a list of references (cited works), each of which is followed by a list 
of the sources (citing works) which cite it. There is no limitation of citations to 
domestic publications or to year of publication. Although the Index . is primarily 
intended to be used as an information retrieval system, other potential uses are 
presented in the report. Counts of citations of published work measure the quality of 
a publication and of an individual’s contribution to science. Data are presented which 
bear on the nature of citation counts and the usefulness of these counts in 
evaluating the statistical evaluation of experimental groups.. The control of these 
groups includes control of the academic fields in which individuals work and publish. 
Although the extensive educational literature would be more difficult to index than 
scientific works, it is felt that the expansion of the Index to include nonscientific fields 
would increase the value of this research tool. (WM) 
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The Use of Publication Citations in Educational Research^ - 

John A. Creager 

•cr 

American Council on Education 

Many educational research studies seek to ascertain the quantity and quality 
of tangible achievements of the subjects. Such information may constitute one 
or more elements of a composite criterion in studies of the outcomes of educa- 
tional selection, types of support, learning processes, or exposures to kinds 

of educational environments. Astin (1964) has stressed the importance of noting 

« 

the ecological nature of criterion variables; from the viewpoint of measurement, 
this implies the need for insight into factors affecting the criterion measure- 
ments. Evaluating tangible achievements may be necessary in selecting persons 
to be admitted to some program, and in ascertaining the outcomes or impacts of 
a program. In any case, the relevance, reliability, and relationships to other 
measurements of the items counted, the methods used to count, and factors affect- 
ing the counts must be explicitly known. The purpose of this report is to examine 
the scope and limitations of citation counts as measures of the influential pro- 
ductivity of scientists, and to suggest that the art in its present state might 
possibly be extended to the nonscientif ic disciplines. 

A number of people reflecting on the limitations of using the number of 
publications as a measure of contribution to science, have suggested that the 
number of citations of published work be used instead. Presumably, many mediocre 
publications are never cited in contrast to seminal works which are cited more 
frequently and for a longer time. Citation counts should therefore represent some 
selection on the basis of quality. By dint of a great deal of effort, Pelz (1955) 

*A. shorter version of this paper will appear in The Journal of Educational 
Measurement . 1967, In Press. 
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was able to compute citation counts as a criterion measure within the field 
of physiology. As a practical matter, however, such computation was far too 
laborious a job to be undertaken in large-scale studies until the appearance 
of the Science Citation Index (Garfield and Sher, -1963) produced by the 
Institute for Scientific Information with support of the National Science 
Foundation. This index makes this kind of data readily available. The 
1963 edition of this index covers a wide range of journals published in 
1961. The 1965 edition covers a still wider range of the 1964 Journals. 

The Science Citation Index 

The basic idea underlying citation indexing is expressed in the fol- 
lowing quotation: "A citation index is an ordered list of references 
(cited works) in which each reference is followed by a list of the sources 
(citing „works) which cite it. The cited work may have been quoted, discussed, 
criticized (as in a book review) etc* "(Garfield, 1965a). The mechanics 
of constructing such an index consist first of examining the journals and other 
source documents published in a given year (e.g., 1961, 1964 or 1965, the 
three source years for which general science citation indexes have been 
published) . The first-named authors of articles in the source documents 
become the units of a source index. Then the authors of each reference 
cited by the source authors in a source document becomes a unit of the 
citation index, no matter what year the cited reference was published. 

Indeed, one can find Aristotle, Galileo, Faraday, and Darwin among the 
cited authors in the 1964 Science Citation Index. This fact also illus- 
trates that there is no restriction of citation to domestic publications. 

The editor of the Index has provided a detailed description of its nature and 
potential uses (Garfield, 1964b); Steinbach (1964) has reviewed the 1961 Index 




and commented on its potential uses and misuses. The Index j s intended primarily 



as an information retrieval system facilitating searches of the literature. 
However, Price (1965) shows its value as a source of information about the 
sociology and history of science. As indicated earlier, the use of such a 
citation index for evaluation purposes has also been suggested, not without 
some misgivings. It is therefore relevant to consider the various factors 
which may affect the evaluative significance of citation counts. 

The meaning of an individual citation is implicit in Garfield's state- 
ment, quoted above, about the basic idea underlying citation indexing. On the 
face of it, a single citation means only that the cited work, whatever its 
year of publication, its authorship, or its quality, was referred to by some- 
one in a publication appearing during the source year. Works may be cited for 
a variety of reasons, but it is plausible to assume that the drudgery of biblio- 
graphic presentation tends to minimize irrelevant listings. Most commonly, an 
author cites another work to connect his work with current relevant activity 
in the field of investigation. This is part of the "sociology of science": 
one has noticed and reacted to a cited work. In this sense, at least, a cited 
work is influential. With the exception of summaries and reviews, articles 
rarely cite works in order to criticize them. Where the purpose in citing a 
reference is primarily to establish connections with related scientific activity, 
it may be presumed that most authors prefer to enhance the prestige of their 
own papers by citing the references of higher technical quality as well as the 
most relevant and the more recent. When a particular reference or a particular 
author is frequently cited, we can presume that this frequency is positively 
correlated with the quality of the cited work. 

Temporal Effects 



Gitation counts fot individual authors are a function of the extent of 
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coverage provided by a given citation index. The coverage in turn is dependent 
upon field variations in publication and reference citing practices. The 
probability of an author's having a citation based on a given source year is 
considerably greater if his publication appeared in years immediately prior to 
the source year. For example, a paper published in 1964 in a journal covered 
by the 1964 Science r.i tat ion Index would be more likely to refer to papers pub- 
lished in 1960-64 than to those published in 1950-54 (Weiss, 1960). For 
reasons disci:issed below, a person may not have published much during the 1960- 
64 period, however extensive and excellent his publications in other periods. 
Conversely, a person may have completed an extensive block of work about 1960, 
submitted several papers in 1961, and had them published in 1963. The implica- 
tion is that counts based on a single source-year index will result in an 
evaluation measurement limited in reliability and validity. The accumulation of 
counts based on several source years will increase the reliability of a 
citations variable. So far, the Science Citation Index has been published 
only for 1961, 1964, and 1965. The publishers hope to fill in the missing 
years, 1962-1963, and to keep up the Index on an annual basis. 

Coverage 

The number of source journals covered by the Science Citation Index_. has 
gradually been expanded. The 1961 Index covered 613 source journals, the 1964 
Index . 700, and the 1965 Index covers about 1000. Similar growth in the cover- 
age may be noted in the number of source articles, the number '"•f citations, and 
the number of authors. The 1961 Index was generally thorough in its coverage 
of journals in biology, chemistry, and physics. Geology journals were not 
included, but are covered in the 1964 Index . Thus, an earth scientist would 
receive 1961 citations only if he had published in the physical science jour- 
nals. Within physics, coverage of astronomy and space science was improved 
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in the 1964 Index . Interdisciplinary journals are fairly well represented. 
Journal coverage is better in science than in technology. Most of these cover- 
age policies, however, are a function of differing publication and referencing 
practices in various fields. For example, citation counts in engineering, 
mathematics, and geology are likely to be low and those in biology, chemistry 
and physics relatively high. Such factors as the scope and heterogeneity of a 
field, the number and size of journals published in given areas, the state of 
historical development as it affects the publication net in a given field, and 
different degrees of encouragement to publish and to relate one's work to 
others' in the network all affect the field differences in citation counts. 

The implication is that, if citation counts are to be used in evaluation 
studies, within-field base rates for citations should be considered and the 
counts must be statistically controlled by field. These considerations along 
with others to be discussed later, warn against evaluative use of individual 
counts for personnel decisions that may affect a person's career. However, 
with proper controls, individual counts may be used as data in statistical 
evaluation of groups. 

Effects of the Career Situation 

Different types of career situations provide different opportunities for 
producing publishable work, getting it published, and then having it cited by 
others. In addition to the field effects mentioned earlier, time factors in 
career progression, type of employer, and on-the-job functions affect publica- 
tion and citation rates. A person's opportunity to publish in the scientific 
journals is extremely limited prior to his attaining the doctorate degree, 
rises to one or more peaks in his early post-doctoral years, and then may drop 
as his professional activity shifts from research to administration. Heavy 
teaching loads as well as administrative activity may limit publication and 
therefore citation rates. Industrial employment may be associated with research 
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not immediately publishable, if publication involves competitive disadvantage 
for the employing company. In government, publication may be restricted to 
classified documents or to agency reports having limited circulation. The 
Science Citation Index has best coverage in the scientific journals for fields 
which emphasize research results. The Index also covers citations of books, 
dissertations, personal communications, and patents, but articles in source 
journals are more likely to refer to other journal articles than to industrial 
and governmental reports. The implication for evaluative use of citation counts 
is that controls are needed to take into account stage of career, employer 
category, and relative amounts of time in various functions (research, teaching, 
administration) . 

What and How to Count 

When the Science Citation Index is used as a basis of evaluative information 
about individuals, various methodological problems arise, some of which have 
been discussed already. For a given individual either the number of items 
cited or the number of citations may be directly counted. From this information 
one may compute the number of citations per cited item, or the ratio of cited 
publications to total publications (obtained from a separate bibliography) . 

Some information concerning the interrelations among these several variables 
will be presented later in this report. Evidence and experience so far 
suggest that, for evaluative purposes, citation counts are preferable to 
counts of cited items or to derived ratios. 

Should all citations be counted? Approximately 8% of the total citations 
in the 1961 Index are self-citations; most individuals never cite themselves; 
a few are cited primarily by themselves. Although extensive self-citation 
appears to be associated with only average or below average performance on other 
indicators of quality- and may be primarily a form of self-advertisement, some 
self-citations are quite legitimate. Not only do some obviously excellent persons 
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(such as Nobel prize winners) do it, but also in a field like mathematics, 
small and relatively isolated groups of scientists, interested and cooperat- 
ing in a new area may be communicating primarily with each other. For most 
educational research purposes, self-citations should probably be excluded 
from the counts, unless it can be shown that they contribute to the intrinsic 
measurement of impact on science. Similar considerations apply to other special 
categories of those referring to private communications, dissertations, patent 
disclosures, and other publications where counting such citations is relevant 
to the aims of a study. None of these categories is likely to amount to an 
appreciable percentage of the total count. 

Multiple Authorship 

The Science Citation Index lists citations by first-named author. In the 
frequent case of multiple authorship, various procedures are used to deter- 
mine the order of listing authors: by alphabet, seniority, direct or indirect 

responsibility for the article, or other means. Obviously these practices 
imply some hazards in using counts for evaluative purposes. However, in 
follow-up studies of applicants for graduate fellowships, a check on citation 
counts for subjects at various positions in the alphabet failed to detect any 
bias from this source. Another check of the bibliographies supplied by question 
naire respondents indicates that multiple authorship is not common in the follow 
up samples, and when it does occur, the subject is usually the first-named 
author. If complete reference information for a subject is available in his 
bibliography, it is theoretically possible to enter the index under the name 
of the senior author and count the number of citations of the articles of which 
the subject is a secondary author. Due to the extensive effort required and 
because not all subjects are consistent in giving complete bibliographic 



information, it may be necessary to accept this as a source of unreliability 
in the citation counts. Nevertheless, fairly reliable information could be 
obtained by more extensive and careful effort, if considerations of validity 
and relevance should justify it. 

Reverse J-Curves 

Among persons whose publications are cited at all, the distribution of 
numbers of persons receiving given numbers of citations is markedly skewed: 
many are cited only once, fewer twice, and so on. Although a count of five 
or six citations based on a given source year is not uncommon, the frequency 
of higher citation counts drops off rather rapidly, more than a dozen or so 
being rather rare. However, in an unselected group of scientists, we find that 
many are not cited at all, even though most of them are in academic situations 
and doing reasearch work. Either they do not publish, or their work is not 
referred to by others. In such a group the skewness is so marked as to be 
essentially a reverse J-curve, the situation typical for former fellowship 
applicants and for the members of a professional society. 

For research purposes, either gross grouping of the counts or nonlinear 
transformations may be required in statistical treatment of the data. Although 
such treatment may result in lower validity coefficients, fine distinctions 
among the heavily cited subjects are of doubtful relevance. 

Evaluative Content 

The significance of citation counts as a measure of one's contribution to 
science may be discovered by counting how many citations are received by persons 
awarded Nobel prizes, or by persons elected to the National Academy of Sciences, 
for a year preceding such a nomination and therefore unaffected by it. 



Although such recognition is partly a reflection of earlier visibility and 
cited achievement, it is an alternative, but not wholly independent, indica- 
tion of recognized scientific achievement and contribution. 

These counts have been made, and show that persons so selected were 
cited, on the average, over twenty times as frequently as were the general 
run of persons cited. As an indicator that citation is a measure of quality 
of an article, it was found that this was not just a function of the fact 
that these people had published much. The articles they authored were also 
cited about twice as frequently as were all other articles that were cited. 

On the basis of such evidence as this, serious study was undertaken of the 
utility of citation counts, and of some derivative measures such as ratios 
of citation counts to number of cited works for former fellowship candidates. 

If citation counts are to be used in selection research for purposes of 
evaluating on-the-job performance and of validating selection variables, it 
is relevant to ask whether any external evidence exists that such counts have 
evaluative content. The 1955 Universtiy of Utah Research Conference on the 
Identification of Creative Scientific Talent included some relevant comments 
by Pelz (1956), referring to an earlier study by Lieberman and Meltzer (1954). 
The study in question examined the Annual Reviews Physiology for three 
years and obtained citation counts on about 1800 members of three major 
physiological societies. A correlation of .50 was obtained between the 
number of publications published during the interval and the number of cita- 
tions. Of course some were cited who had not published in this particular 
period, but the mean number of publications by each subject was about six 
and the mean number of citations, three. As Pelz commented, 
tive process is operating, presumably in terms of quality." 



. .a selec- 



- 10 - 



Nobel Laureates 

The publishers of the Science Citation Index, although cautious about 
the evaluative use of the Index , counted citations in the 1961 Index for 
several Nobel prize winners , What is important to realize here is that 
the citations were made prior to the awarding of the Nobel prize, and are not, 
therefore, attributable to publicity from the prize itself. Whereas the base- 
line number of items cited per author was 3.37 in the 1961 Index , the Nobel 
winners in physics, chemistry, and medicine for 1962 and 1963 averaged 58.10 
cited items. The average number of citations was 5.51 for the total file and 
169.00 for the Nobel winners. The number of citations per cited item was 
nearly twice that expected from the base rate. 

Members of the National Academy 

A count was made of the citations in both the 1961 and the 1964 Indexes 
for 35 persons elected to membership in the National Academy of Sciences in 
April, 1965. Election to the National Academy of Sciences is generally 
regarded as a mark of distinction based on original and extensive contribu- 
tion to science and technology. All 35 had received one or more citations, 
and only four of the 35 had fewer than six citations. These four were persons in 
either fields or functions where publication and citation is less likely. Most 

electees had counts that were very high, in the 20-100 range, and several had 
more than 100. 

Former Applicants for Graduate Fellowships 

Various kinds of reliability and validity data bearing on the evaluative 
content of citation counts were obtained in follow-up studies of former 
applicants for National Science Foundation graduate fellowships. The details 
of these studies provide the sampling information and context for evaluating 
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citation counts (Creager, 1962a, 1962b, 1965; Creager and Harmon, 1966) . One 
study, based on terminal applicants, that is, those in final stages of 
graduate education in 1952-53-54, provided the basis for a pilot study of 
citation counts. Most had finished graduate study and had published in 
time to be cited in the 1961 and 1964 Indexes. Further validation data were 
obtained from the first-year and intermediate applicants of 1955 and 1956, 
using counts based on the 1965 Index. First year applicants seek fellow- 
ship support for the first year of graduate study. Intermediate applicants 
have already been admitted to graduate study. 

Fifty percent of the 1506 men and 48% of the 48 women in the terminal 
group received one or more citations. Only 167 0 of the 37 nondoctorates 
received citations. In two groups, rated "high" and "low" in on-the-job 
performance, 68% of the "high" group and 41% of the "low" group received 
citations. The percentages of the men receiving one or more citations in 
the 1961 Index are presented in Table 1. 



These data may be interpreted in terms of known field differences, in 
doctorate attainment rates, publication practices, and employer categories. 
On this basis, these data indicate that doctorate attainment and academic 
employment with opportunity to do research in an experimental science are 
favorable conditions for receiving citations. The low count in geology is 
attributable in part to lower publication rates, and in part to lack of 
geology journals in the 1961 Index . The lower frequency of publication in 
mathematics largely accounts for the low percentage in that field. 
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Table 1 



Percentage of Fellowship Applicants Receiving One or More Citations 



Field 


Number of cases 


Percentage cited 


Biology 


380 


58 


Chemistry 


457 


52 


Engineering 


140 


21 


Geology 


80 


15 


Mathematics 


117 


29 


Physics (including astronomy) 


301 


69 


Psychology 


31 


45 
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Other Citation Variables 

In the same study, four variables in addition .to the number of citations 
were considered. One of these, the number of cited references, was also 
obtained by direct count from the Science Citation Index . The other three 
are ratios: number of citations per cited reference; references cited divided 
by an index of productivity; and number of citations divided by the pro- 
ductivity index. A high citation count may result from infrequent citing of 
each of many publications, or from frequent citing of a few publications. A 
count of the number of cited reference ignores this distinction, but the 
ratio of citations to references points it up. The other two ratios attempt 
to reduce the effect of total productivity: i.e., to emphasize the quality 
rather than the quantity content of the citations information. 

The intercorrelations among the five criterion elements derived from the 
Citation Index and their correlations with the productivity index provide 
further information about the meaning and potential utility of these elements 
in evaluation research. Such intercorrelations were obtained rn five of the 
fields. Since the various fields have different patterns of relations between 
production and citation, some variations in these correlations could be 
expected, and were, in fact, found. However, the general pattern of inter- 
correlations may best be illustrated by the typical intercorrelation values 
obtained among the six variables® Median values computed from the five field 
samples are presented in Table 2. 
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Table 2 



Intercorrelations Among Citation and Productivity Variables 



Variable 


1 2 3 4 5 6 


1. Number of References 


1.00 


2. Number of Citations 


.96 1.00 


3. Citations/References 


.75 .85 1.00 


4. Raf erences/Productivity 


.85 .83 .55 1.00 


5. Citations/Productivity 


.82 .83 .71 .90 1.00 


6. Productivity Index 


.33 .30 .22 .10 .08 1.00 



Since these correlations were obtained in the terminal fellowship appli- 
cant groups, they may be somewhat lower than those which would be obtained 
from a more general sample of scientists or former fellowship applicants. 

This is known to be the case for the correlation between productivity and 
citations. The report by Pelz (1956) gave a correlation of .50, a figure 
typical of those later obtained in the first-year and intermediate applicant 
samples. The correlation of .96 between counts of citations and counts of 
cited references indicates that these two variables are probably inter- 
changeable, despite the logical difference in their meaning. The ratios 
have the disadvantage of having to be computed for each individual; this 
effort can be justified only if the ratios can be shown to have higher 
relevance, reliability, or validity in the context of a given study. 

Validities 

Validities of fellowship selection variables (Graduate Record Examination 
scores, undergraduate science grade average, and faculty ratings of over-all 
scientific ability) were computed against the five variables derived from 
citations information. This was done separately in five fields: biology, 
chemistry, engineering, mathematics, and physics. Neither job function nor 
employer category were controlled in this pilot study. A composite of selec- 
tion variables predicted the citation counts with a validity , of about .25. 
With the exception of biology, where the ratios were slightly better pre- 
dicted, the number of citations was most predictable. Mean faculty ratings 
proved the best predictor with a little help from the Advanced GRE Test score. 

It is of special interest that the citations are more predicatable by a 
few points, and uniformly so, than productivity. Correlations of citation 
counts with on-the-job performance ratings lie in the .25 to .35 range. 
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Thus it seems. likely that citation counts contain some variance that measures 

quality as well as quantity of output. 

Enriching the Picture 

In the second and more thorough follow-up study involving first-year and 
intermediate fellowship applicants, the distributions and predictability of 
counts were again studied. In view of the results of the study of the termi- 
nal samples, only counts of citations by others were made; no counts of cited 
references and calculations of ratios were made. In this study, citation 
counts were obtained for all subjects (99%) for whom there was no substantial 
doubt concerning the matching between fellowship records and the citation 
indexes. Thus, information about nonrespondents to follow-up questionnaires 
could be gathered, which illustrates an important advantage of the citation 
counts: their dependence of follow-up by questionnaires and of on-the-job 

performance reports. Bibliographic information from the questionnaire is 
helpful, however, in ensuring that the cited references are, indeed, those 

of the subject. 

In this study, citation counts were obtained primarily from the 1964 
rather than the 1961 Index . This choice was indicated partly by the 
greater likelihood that these subjects would have completed graduate study 
and achieved citable publications by 1964, and partly by the 1964 Indexes 
more extensive coverage of the, literature. However, it is theoretically 
possible to increase the relevance, reliability, and validity of the counts 
if they are based on more than one source year. Therefore, in biology, 
chemistry and physics separate counts were made based on the 1961 Index , and 
the counts from the two sources were combined. Since only a very small pro- 
portion of the subjects had completed graduate study by 1960, the counts 
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based on the 1961 Index were much lower than those based on the 1964 Index. 
Otherwise, the distributions of counts from the two source years are similar. 
The correlations between 1961 and 1964 counts are typically in the .45 to .60 
range and are higher for intermediate applicants and for 1955 applicants since 
these groups are more likely to have completed graduate study in time for 
1961 counts to be meaningful and reliable. Validities in the various samples 
were uniformly higher for the 1964 counts. However, inclusion of the 1961 
counts increased the validities slightly in several of the samples. It is 
apparent that combining counts from several source years can increase reli- 
ability, provided that the subjects have had time to progress in their careers 
far enough to have completed and published some research. Since this is not 
uniformly the case for the subjects in the study, distributions and validities 
will be presented only for the counts based on the 1964 Index* 

Distributions of Citations 

Distributions of citation counts for the subjects in the present study 
are presented in Table 3, which is divided into four horizontal sections 
representing various analytic groups of 3504 subjects. In the top section, 
subjects are grouped by application year, level, and sex. In the second 
section, they are grouped by field; in the third section, by questionnaire 
return status and employment categories. Finally the distributions for 
awardees and nonawardees are present. All distributions show the strong 
reverse-J curve previously noted for groups which include persons receiving 
no citations. In addition it is found that those groups with larger pro- 
portions of individuals having no citations also have smaller proportions of 
citatianrreceiving subjects with high citation rates. Not surprisingly, 
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Table 3 

Distribution of Counts in 1964 Science Citation Index, in Follow-up Samples 



r * ‘ 

Sample 






Number of Citations 


Uncer- 

tain 

match 


Subjects 
with one 
or more 
citations 


Sample 

size 


None 


1 


2 


3 


4 


5 


6-8 


9+ 


All subjects combined 


N 


3504 


2315 


292 


197 


118 


84 


70 


134 


259 


35 


1154 




% 


100.0 


66 . 1 


8.3 


5.6 


3.4 


2.4 


2.0 


3.8 


7.4 


1.0 


32.9 


1955 1st year men 


N 


844 


564 


59 


56 


24 


21 


11 


38 


63 


8 


272 




% 


100.0 


66.8 


7.0 


6 . 6 


2.8 


2.5 


1.3 


4.5 


7.5 


0.9 


32.2 


1955 1st year women 


N 


128 


113 


6 


5 


0 


1 


2 


0 


1 


0 


15 




% 


100.0 


88.3 


4.7 


3.9 


0.0 


0.8 


1.6 


0.0 


0.8 


0.0 


11.7 


1956 1st year men 


N 


935 


659 


75 


47 


26 


19 


17 


28 


54 


10 


266 




% 


100.0 


70.5 


8.0 


5.0 


2.8 


2.0 


1.8 


3.0 


5.8 


1.1 


28.4 


1956 1st year women 


N 


150 


137 


6 


3 


1 


1 


1 


1 


0 


0 


13 




% 


100.0 


91.3 


4.0 


2.0 


0.7 


0.7 


0.7 


0.7 


0.0 


0.0 


8.7 


1955 intermediate 


N 


771 


426 


79 


45 


40 


24 


22 


43 


85 


7 


338 


men 


% 


100.0 


55.3 


10.2 


5.8 


5.2 


3.1 


2.9 


5.6 


11.0 


0.9 


43.8 


1955 intermediate 


N 


78 


62 


7 


1 


2 


1 


1 


2 


2 


0 


16 


women 


% 


100.0 


79.5 


9.0 


1.3 


2.6 


1.3 


1.3 


2.6 


2.6 


0.0 


20.5 


1956 intermediate 


N 


553 


311 


59 


39 


25 


17 


16 


22 


54 


10 


232 


men 


% 


100.0 


56.2 


10.7 


7.1 


4.5 


3.1 


2.9 


3.9 


9.8 


1.8 


42.0 


1956 intermediate 


N 


45 


43 


1 


1 


0 


0 


0 


0 


0 


0 


2 


women 


% 


100.0 


95.6 


2.2 


2.2 


0.0 


0.0 


0.0 


0.0 


0.0 


0.0 


4.4 


Biology 


N 


712 


443 


60 


41 


25 


20 


15 


27 


71 


10 


259 




% 


100.0 


62.2 


8.4 


5.8 


3.5 


2.8 


2.1 


3.7 


10.0 


1.4 


36.4 


Chemistry 


N 


749 


434 


58 


55 


41 


26 


21 


34 


74 


6 


309 




% 


100.0 


57.9 


7.7 


7.3 


5.5 


3.5 


2.8 


4.5 


9.9 


0.8 


41.3 


Engineering 


N 


585 


462 


48 


25 


10 


8 


5 


12 


7 


8 


115 




% 


100.0 


79.0 


8.2 


4.3 


1.7 


1.4 


0.9 


2.0 


1.2 


1.4 


19.6 

| 


Geology 


N 


177 


144 


12 


6 


5 


1 


2 


1 


0 


6 


33 




% 


100.0 


81.4 


6.8 


3.4 


2.8 


0.6 


1.1 


0,6 


0.0 


3.4 


18.6 


Mathematics 


N 


319 


272 


20 


6 


6 


3 


2 


2 


6 


2 


45 




% 


100.0 


85.3 


6.3 


1.9 


1.9 


0.9 


0.6 


0.6 


1.9 


0.6 


14.1 


Physics 


N 


800 


434 


80 


60 


27 


26 


23 


55 


87 


8 


358 




% 


100.0 


54.3 


10.0 


7.5 


3.4 


3.3 


2.9 


6.9 


10.9 


1.0 


44.7 


Psychology 


N 


162 


126 


14 


4 


4 


0 


2 


3 


8 


1 


35 




% 


100.0 


77.8 


8.6 


2.5 


2.5 


0.0 


1.2 


1.8 


4.9 


0.6 


21.6 


Employed by academic 


N 


953 


518 


85 


59 


41 


32 


31 


59 


128 


0 


435 


institution 


% 


100.0 


54.4 


8.9 


6.2 


4.3 


3.4 


3.3 


6.2 


13.4 


0.0 


45.6 


Employed by business 


N 


603 


406 


49 


29 


27 


10 


10 


23 


48 


1 


196 


or industry 


% 


100.0 


67.3 


8.1 


4.8 


4.5 


1.7 


1.7 


3.8 


8.0 


0.2 


32.5 


Employed by all levels 


N 


232 


130 


19 


24 


14 


7 


5 


13 


20 


0 


102 


of government 


% 


100.0 


56.0 


8.2 


10.3 


6.0 


3.0 


2.2 


5.6 


8.6 


0.0 


44.0 


Other employment 


N 


182 


168 


11 


0 


0 


1 


0 


0 


2 


0 


14 


categories 


% 


100.0 


92.3 


6.0 


0.0 


0.0 


0.5 


0.0 


0.0 


1.0 


0.0 


7.6 


Questionnaire 


N 


1534 


1093 


128 


85 


36 


34 


24 


39 


61 


34 


407 | 


non-returnees 


% 


100.0 


71.3 


8.3 


5.5 


2.3 


2.2 


1.6 


2.5 


4.0 


2.2 


26.5 


Awardees 


N 


943 


500 


96 


65 


45 


26 


19 


59 


122 


11 


432 




% 


100.0 


53.0 


10.2 


6.9 


4.8 


2.8 


2.0 


6.3 


12.9 


1.2 


45.7 


Non-awardees 


N 


2561 


1815 


196 


132 


73 


58 


51 


75 


137 


24 


722 




% 


100.0 


70.9 


7.7 


5.2 


2.9 


2.3 


2.0 


2.9 


5.3 


0.9 


28.2 
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larger proportions of subjects with one or more citations are found among the 
older persons: the 1955 and intermediate applicants as compared with the 

1956 and first-year applicants. In the case of differences between first- 
year and intermediate applicants some selection due to academic attrition 
has also occurred. In these otherwise unselected groups, citations are 
very much more frequent among men. 

Field and Employer Variations 

Citation rates are higher in biology, chemistry, and physics than in 
other fields, confirming the results observed with the terminal subjects. 

These groups are more often employed in the academic world and so have 
more opportunity for publishable and citable research activity. In this 
respect government employment is also favorable, although the proportion of 
subjects receiving many citations is higher in the academic group. Although 
the chances of being cited are lower for those employed by industry, about 
one-third of the present sample so employed received citations. Evidence 
presented elsewhere (Creager, 1962, 1965) indicates that a greater proportion 
of the lower ability applicants, as rated by the fellowship evaluation panels, 
are absorbed by industry in positions of lower technical responsibility. The 
reason for the very low citation ratings in the "other employment" category 
is that most of these people were unemployed or working in nonscientif ic jobs. 



Quality Group Predicts Citations 

In view of the earlier evidence for quality variance in citation counts 
(e.g., the higher citation rates for Nobel prize winners and electees to 



the National Academy of Sciences) the markedly higher percentages receiving 
citations among fellowship awardees (46%) than among nonawardees (28%) can be 
taken as further evidence of successful panel evaluation of fellowship appli- 
cants in terms of rated ability. The panel evaluation results in placing each 
applicant into one of six "Quality Groups," with Group I consisting of those 
considered most qualified to receive a fellowship. The distribution of 
citations received in the 1964 Index by the panel-determined quality group 
is of interest: 



Quality Group: 


I 


II 


III 


IV V VI 


% With one or more 
Citations (1964) 


48.1 


43.0 


36.1 


31.3 21.8 12.5 



The correlation is clear though far from perfect. 

Ph.D.s Get Cited More 

It is not surprising that citation and attainment of the doctorate are 
related, as shown in Table 4 where frequencies and percentages of subjects 
in the total sample are cross-tabulated in terms of doctorate attainment and 
cited versus not-cited categories. One is rather unlikely to be cited if 
he has not attained the doctorate. Nevertheless a small group of nondoctorates 
were cited. The restriction of citation indexing to first-named author and 
the greater likelihood that published research in experimental science will 
be under the leadership of a doctorate-holder lead one to expect the positive 
relationship between doctorate and citations. The nondoctorates who are 
cited include those cited for work done during graduate study, and a few 
excellent and enterprising individuals who left graduate school and are in 
situations favorable for publishable research. More interesting is the 
large group of doctorate-holders without citations. This group includes 
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capable people not in research settings and unproductive people who neverthe- 
less manage to complete doctorates. Many of those not cited attained the 
doctorate too recently to have published articles that could be mentioned in 
the 1964 Index . Except for this, citation counts could compensate in part 
for imperfections in criteria of doctorate attainment if both kinds of 
criteria are included in a followup composite criterion. The point-biserial 
correlations between the doctorate attainment dichotomy and the citation 
counts typically lie between .30 and .40 in the various samples used in 
validation. 



Time Spent in Research 

In several of the foregoing sections, there has been frequent mention 
of the amount of time spent in research activity that might lead to publish- 
able and citable results. The present data provide an opportunity to examine 
this question quantitatively for 1831 respondents who answered the question- 
naire item regarding time spent in various functions on the job. Information 
supplied by the respondent was coded into nine patterns defined by percentage 
of time spent in teaching, research, administration, and other activities. 
Table 5 presents the number and percentage of subjects in each functional 
pattern who have one or more citations in the 1964 Index . 
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A, 

B 



D 



G 



Table 5 



Citations as a Function of Time Distribution 



Activity Pattern 



99- 1007c time in research 6c development 



60-987o time in research 6c development 
with remainder in administration 



60-987o time in research & development with 
remainder in teaching and/or other 



50-50 split in time between research- 
development 6c teaching activities 



507o time in research 6c development with 
remainder in various combinations of 
teaching, administration, and other 
activities . 



More than 507 o time teaching with some 
research 6c development 



More than 507 o time teaching with some 
administration 



H 



M 0 re than 507 o time teaching with remainder 
in various combination of research, 
administration, and other activities 




Total 




nan 



Number 

of 

subjects 



419 



188 



272 



109 



137 



108 



94 

136 



1831 



Subjects with one 
or more citations 



Number 



192 



96 



161 



64 



60 



25 



10 



40 



740 



Percent 



45.8 



51.1 



59.2 



58.7 



43.8 



23.1 



10.6 

29.4 



40.4 
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These data support the expectation that those with more time spent in 
research and development are more likely to receive citations. However, the 
relationship is not uniform; proper interpretation requires taking account 
of employer category and of the proportions of persons with high ability 
associated with the various functions. The supplemental data for these 
considerations are presented in Table 6. 



If time spent in research were the main determinant of citation rate, 
Pattern A, in which full time is spent in research and development, would 
have the highest citation rate. The proportion of Pattern A subjects at 
the higher ability levels is close to that for the total sample, but a 
much higher proportion is employed in industry and government where results 
are more likely to be disseminated in house publications, technical memoranda, 
patent disclosures, and administrative reports. This picture is even more 
striking in Pattern B, where research functions are primary but where there 
is also appreciable administrative activity. Here the citation rate is 
higher, but the proportion of persons who have high-level ability is the 
same. It is likely that many of these people are in research administra- 
tion in contrast to business administration; they are in a position to 
summarize and selectively disseminate industrial and governmental research 

results in the citations source journals. 

Concerning the other patterns, it may be noted that a greater pro- 
portion of the high-level people appear in academic settings and spend at 
least half of their time in research (e.g. Patterns C and D) . These people 
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Table 6 



Employer Categories and Quality Group Level by Time Distribution Pattern 








Percentage of cases in given time 
Pattern Category who are found in: 




Time percentage distribution pattern 


Academic 

jobs 


Industrial 

jobs 


Gov ' t 
jobs 


Quality 
Groups 
I or II 


A. 


99 - 1007 o time in research & developin' t 


26 


51 


23 


30 


B. 


60-98% time in research & development 
with remainder in administration 


16 


61 


23 


29 


C. 


60-98% time in research 6c development 
with remainder in teaching &/ or 
other 


64 


22 


14 


40 


D. 


50-50 split in time between research- 
development & teaching activities 


100 


0 


0 


43 


E. 


507o time in research 6c developin' t 
with remainder in various combina- 
tions of teaching, administration, 
6c other activities 


64 


25 


11 


34 


F. 


More than 507, time teaching with 
some research 6c development 


100 


0 


0 


23 


G. 


More than 507, time teaching with 
some administration 


98 


2 


0 


15 


H. 


More than 507, time teaching with 
remainder in various combina- 
tions of research, administration 
6c other activities 


100 


0 


0 


31 


1 . 


All other combinations 


34 


52 


14 


27 




Total 


53 


34 


13 


31 
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have high citation rates. It does not appear that some teaching load is 
detrimental to achieving citable research publications; indeed, it may be 
favorable for those with the ability to do both and for those in a position 
to relate the two functions in their professional life. This is more likely 
to be the case in the large doctorate-granting institutions. Those in smaller 
institutions, and primarily in teaching positions, at the undergraduate level, 
have much lower citation rates. Among former fellowship applicants, the 
proportion of the high-ability applicants who arrive in this combination of 
employment and function is relatively small. Finally, persons in the last 
pattern, who spend approximately equal portions of time in a variety of 
activities include the technician-level personnel, especially useful in 
industry and in other settings where carrying out detail is required more 
than is broad planning and supervision of research. 

Correlations of Citations 

To supplement the evidence obtained from distributions of citation 
counts for various groups of scientists and from other factors affecting 
citation counts, correlational analyses were undertaken. The correlations 
of citations with other follow-up criterion variables are presented in 
Table 7. The ability of the several predictor measures, separately and 
in the Summary Score composite of selection variables, to predict citation 
counts and the productivity index is shown in Tables 8 and 9. The 

correlations of Quality Group with these two criteria are also shown. 

The previously noted relationship between citation counts and doctorate 
attainment and that between counts and productivity are well illustrated 



in Table 7 . 



Correlations with overall rating and with income tend to be 
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Table 7 

Correlations Between 1965 Citation Counts and Other Criterion Elements 



Sample 


N 


Ph. D. 
attainment 


Overall 

rating 


Income 


Productivity 

index 


1st year - 1955 


427 




40 


27 


07 


52 


1st year - 1956 


471 




37 


26 


13 


45 


Intermediate- 19 55 


409 




24 


31 


21 


49 


Intermediate- 19 56 


282 




_23 


25 


11 


42 


Academic - 1955 


437 




24 


33 


24 


49 


Academic - 1956 


442 




24 


28 


22 


44 


Industrial-1955 


293 




40 


20 


17 


49 


Industrial-1956 


CM 

CM 




39 


11 


14 


_54 


Governmental- 19 55 


106 




36 


2 7 


19 


48 


Governmental- 1956 


100 




43 


21 


27 


30 


Biology - 1955 


182 




26 


29 


: 29 


38 


Biology - 1956 


160 




31 


28 


35 


36 


Chemistry - 1955 


185 




29 


46 


18 


57 


Chemistry - 1956 


171 




32 


40 


26 


57 


Engineering 55+56 


249 




32 


18 


08 


45 


Geology 55 + 56 


109 




19 


25 


01 


3 7 


Mathematics 55+56 


134 




25 


_37 


15 


41 


Physics - 1955 


192 




38 


3 7 


30 


62 


Physics - 1956 


160 




38 


22 


20 


50 


Psychology 55+56 


73 




08 


27 


31 


44 



Note. - Decimal points have been omitted. Coefficients significant at 
the 1% level are underlined; those significant at the .1% level are double- 




underlined 
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Table 8 

Correlations of Income, Productivity, and Citations with Five Predictor Variables 



Sample 


N 


■ — — — — 

Verbal 


Quantitative 


Advanced 


Grade point 
average 


Reference 
rating average 






In a 


Pr 13 


Ci c 


In 


Pr 


Ci 


In 


Pr 


Ci 


In 


Pr 


Ci 


In 


Pr 


Ci 


1st year 

1955 

1956 


427 

471 


-02 

09 


14 

02 


21 

JJ 


22 

2J 


03 

M 


18 

22 


14 

18 


M 

23 


22 

22 


03 

03 


13 

14 


16 

10 


00 

06 


11 

M 


14 

06 


Intermediate 

1955 

1956 


409 

282 


-01 

07 


00 

01 


12 

02 


23 

13 


03 

-02 


18 

05 


10 

13 


01 

07 


10 

15 


05 

01 


07 

-06 


09 

05 


08 

05 


22 

05 


12 

09 


Academic 

1955 

1956 


'437 

442 


-01 

09 


09 

L3 


21 

TO' 


21 

21 


06 

08 


21 

12 


14 

iz 


10 

22 


22 

22 


04 

03 


01 

-03 


07 

00 


09 

03 


13 

09 


11 

04 


Industrial 

1955 

1956 


293 

111 


10 

_20 


13 

06 


16 

05 


20 

11 


00 

00 


09 

02 


11 

11 


22 

11 


14 

18 


12 

04 


09 

03 


14 

11 


07 

09 


01 

04 


12 

08 


Government 

1955 

1956 


106 

100 


02 

00 


-08 

03 


18 

14 


14 

38 


03 

09 


22 

21 


12 


04 

12 


17 

M 


-03 

03 


-02 

02 


16 

16 


08 

14 


03 

07 


19 

07 


Biology 

1955 

1956 


182 

160 


06 

11 


08 

22 


25 

2 


23 

20 


14 

20 


29 

24 


08 

17 


18 

19 


12 

22 


-05 

04 


08 

-01 


12 

07 


01 

13 


04 

08 


04 

15 


Chemistry 

1955 

1956 


185 

171 


-10 

03 


12 

07 


30 

TJ5 


05 

-01 


12 

10 


25 

taamm 

17 


20 

11 


27 

1 


22 

22 


-06 

-07 


05 

04 


13 

11 


00 

07 


15 

14 


22 

22 


Engineering 


249 


05 


08 


09 


06 


04 


02 


03 


11 


13 


10 


08 


04 


10 


08 


05 


Geology 


109 


14 


14 


11 


03 


21 


15 


11 


22 


08 


00 


03 


07 


05 


05 


08 


Mathematics 


134 


16 


26 


16 


29 


13 


09 


17 


21 


20 


-24 


05 


05 


-19 


19 


20 1 


Physics 

1955 

1956 


192 

160 


-05 

02 


21 

04 


18 

05 


07 

17 


24 

13 


30 

12 


09 

22 


22 

22 


11 

18 


00 

-06 


14 

10 


14 

01 


10 

-02 


IQ 

10 


22 I 

-07 1 

|l 


Psychology 


73 


-12 


01 


-03 


-03 


12 


23 


09 


03 


22 


-11 


04 


-13 


00 


09 


-04 I 



IN O L 6 • - UeCimaJL pUJLULS IlclVC UUCll kjvua . LLCU • 

underlined; those significant at the .1% level are double-underlined. 
a Income 

^Productivity Index 
c Citations Counts (1964) 
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Table 9 

Correlations of Income, Productivity, and Citations with 
Summary Score and Quality Group. 




Note.- Decimal points have been omitted. Coefficients significant 
at the 1% level are underlined; those significant at the .17« level 
are double-underlined. 

a Income 

^Productivity Index 
c Citations Counts (1964) 
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higher in those categories where the "publish-or-perish" philosophy is more 
likely to prevail. Correlations with ability measures made several years 
before extensive publication are consistent with the hypothesis that 
citation counts have evaluative content. In groups where the counts are 
more reliable, the correlations with citations are higher than those with 
productivity . 

Summary and Conclusions 

The availability of the Science. Citation Index provides those engaged 
in educational research with a useful tool. Counts of citations of published 
work not only measure quantity of publication, but contain a qualitative 
factor, often preferable in evaluation studies. This report has presented 
data bearing on the nature of citation counts, with special reference to 
potential hazards that arise from several factors that can affect them. In 
spite of these hazards, the usefulness of the counts in statistical evaluation 
of groups can be recommended where awareness of the hazards is accompanied 
by adequate controls in the design and execution of the study. Definition 
and control of experimental groups must include control of the academic 
fields in which subjects work and publish. Many other factors affecting 
citation counts are somewhat, but imperfectly, correlated with field. It 
is therefore highly desirable that additional controls for employment 
situation effects and for other factors which influence the counts be considered 

in the plan and execution of an evaluation study. 

It may well be that the more able persons are more apt to go into those 
employment categories where productive and citable research is most likely. 

Data presented in Table 4 tend to support this assumption. To the extent 
that this is the case, the lower citation counts in some employer categories 
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are not attributable to situational opportunity, but are partly a function 
of ability. Further research is needed to clarify the need for control of 
such selective input to the situations in which citation counts are being 

obtained for their evaluative content. 

Where possible, it is desirable to build up the reliability of citation 

counts by basing the counts on more than one source year. In any case the 
decision to employ citation counts in an evaluation study must be based on 
their relevance to the aims of the study and the possibility of incorporat- 
ing the necessary controls. Extra caution in using citation counts is 
required when the counts are to be used as a selection variable, since 
individual careers may be affected. In such cases considerably tighter 
control of the factors affecting the counts must be exercised than in the 
ex post facto evaluation of groups in a follow-up study. 

Similar cautions are noted by Bayer and Folger (1966) in their study 
of some correlates of citation counts for 467 U.S. biochemistry doctorates. 

This study reports low but significant correlations between citation counts 
and quality of graduate education, and provides additional relevant references. 

With these various precautions in mind, there would seem to be no reason 
why insightful use of this tool should be denied to educational research. 

Since published citations are available only in the science fields, the use 
of citation counts is necessarily limited at present to studies involving 
those fields. The extensive educational literature, as well as that in the 
arts and humanities journals would be somewhat more difficult, but not 
impossible, to index. If citation indexes in the nonscience fields are sup- 
ported and thereby become available, the value of this research tool will be 
greatly enhanced. 
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